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Executive summary 


The project 


The ‘Literacy Octopus’ Dissemination Trial aimed to test the impact on pupil outcomes of disseminating 
research summaries and evidence-based resources to schools. The materials aimed to support 
teaching and learning of Key Stage 2 literacy and were created by leading organisations with experience 
of engaging schools in evidence use. The EEF funded this alongside a smaller trial which included more 
proactive support from these organisations to learn about the impact on both attainment and, in the 
smaller evidence-based support trial, on teachers’ use of research. 


The materials included: 


e an evidence-based printed booklet /mproving Reading: A Guide for Teachers (CEM, part of 
Durham University); 

e printed copies of Better Evidence-based Education magazine, fortnightly ‘Best Evidence in 
Brief’ email summaries, and access to a searchable database, Evidence 4 Impact (IEE, based 
at the University of York); 

e a link to the archived webinar and materials relating to a conference on research evidence 
relating to KS2 literacy (ResearchEd, in collaboration with NatCen); and 

e a subscription to the Teaching How2s website which focuses on evidence-based techniques 
and step-by-step visual guides for teachers (delivered by the Campaign for Learning in 
partnership with Train Visual). 


A sample of 12,500 primary schools was identified by the evaluator, NFER, and randomised into five 
groups of 2,500. In spring term 2015, each of the four providers sent evidence-based materials to their 
2,500 primary schools. The remaining 2,500 schools were in the control group. Given the trial did not 
require any active engagement or consent from the schools, the materials were simply sent to the 
schools with a brief note about the trial. The evaluation did not aim to collect information about schools’ 
use of the materials, but to provide a large-scale study of the impact of disseminating literacy-related 
evidence-based materials on pupils’ outcomes in KS2 English in summer 2016. 


The trial was funded by the Education Endowment Foundation (EEF), the Department for Education, 
and the Mayor’s London Schools Excellence Fund as part of a round of funding exploring Research 
Use in Schools. It was accompanied by Evidence-based Literacy Support—the ‘Literacy Octopus’ Trial, 
which tested the impact of similar interventions when accompanied by more pro-active support. 


Ceo) aeg (UY (e) ak 


1. The project found no evidence that Literacy Octopus passive dissemination interventions 
improved pupils’ Key Stage 2 English scores compared with the control group. The five padlock 
security rating means we have very high confidence in this result. 


2. These findings suggest that simply disseminating research summaries and evidence-based 
resources to schools is not an effective way for research organisations to support schools to 
improve pupil outcomes. 


3. Itis likely that these materials formed a small part of the information received by schools during 
this time. It is possible that if schools had support to navigate and prioritise that information, 
greater impact could be achieved. Alternatively, schools may need more support in transforming 
such materials into actual change in the classroom. The accompanying Evidence-based 
Literacy Support—the ‘Literacy Octopus’ Trial helps us to assess this. 


4. The evaluator will analyse the following year’s Key Stage 2 results from these schools to explore 
if there is a longer-term effect of the interventions. 
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EEF security rating 


These findings have very high security. The trial was a large-scale effectiveness trial involving a sample 
of 12,500 primary schools—over 80% of the primary school population. It was well-powered, and since 
the primary outcome was obtained from the National Pupil Database, the proportion of data missing 
was low: just 1% of schools, and 6% at the pupil level. Pupils in schools in each intervention arm were 
similar to those in control schools. 


Additional findings 


There was no evidence that any of the Literacy Octopus passive communication approaches impacted 
on pupil attainment in literacy. This was the case for all pupils and for the sub-group of pupils who have 
ever been eligible for free school meals. 


The extent to which schools engaged with or used the materials in this dissemination trial is not known 
as it was not monitored. However, we can say from this trial that attempting to reach a large number of 
schools and teachers through fairly light-touch and inexpensive approaches does not impact in a 
measurable way on pupil attainment outcomes in the year following the dissemination activities. This is 
not to say that schools did not use or engage with the materials in any way: we know from the 
accompanying Evidence-based Literacy Support—the ‘Literacy Octopus’ Trial that just under three- 
fifths of schools did engage, at least to some extent, in passive-arm activity. However, any engagement, 
or otherwise, did not result in impacts on pupil attainment. 


The findings show that even where materials are designed to be evidence-informed and practical, as 
was the case in all of the Literacy Octopus providers’ materials (for example, they were well structured 
and included examples for practice), simple passive dissemination is insufficient to make a difference 
to learning outcomes. 


The Literacy Octopus materials would likely have been additions to the range of research and 
commercial information received by schools, particularly by headteachers—the recipients of 
communication in this trial. Schools and heads may need support to navigate this information and make 
best use of it. 


Follow-up analyses will be conducted in 2017/2018 of the 2017 KS2 English scores, with an addendum 
report to be published in 2018. 


Cost 


All of the materials were inexpensive to provide at scale. Costs to schools in terms of time to engage, 
or not, were not monitored on this dissemination trial. However, we do know from the Literacy Octopus 
evidence-based literacy support trial that it was inexpensive for schools to engage with the materials 
provided, though they did require time to implement any practices suggested by the materials. That 
report provides further details about the per-school and per-pupil costs. 
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Table 1: Summary of impact on primary outcome (Key Stage 2 English) 


CEM Passive Arm 
1 vs control 


IEE Passive Arm 2 
vs control 


ResearchEd 
Passive Arm 3 vs 
control 


How2s Passive 
Arm 4 vs control 


CEM Passive Arm 
1 everFSM vs 
control everFSM 


IEE Passive Arm 2 
everFSM vs 
everFSM control 


ResearchEd 
Passive Arm 3 
everFSM vs 
everFSM control 


How2s Passive 
Arm 4 everFSM vs 
everFSM control 


Effect size 
(95% 


(ore) ayite(=yalere) 
interval) 


0.01 
(-0.01, 0.03) 


0.01 
(-0.01, 0.03) 


0.01 
(-0.01, 0.03) 


0.01 
(-0.01, 0.03) 


0.00 
(-0.03, 0.03) 


-0.01 
(-0.04, 0.02) 


-0.00 
(-0.03, 0.03) 


0.00 
(-0.03, 0.03) 


Estimated 
months’ 
progress 


Literacy Octopus Dissemination Trial 


I Comme) | 
pupils* 


88,088 


86,742 


86,155 


87,701 


P value** 


EEF 
security 
rating 


“Number of pupils refers to the number of pupils’ results analysed in the primary outcome multi-level 


EEF cost 
eclilale) 


ELELE 


ELELEE 


ELLEL 


ELELCE 


model. Note, the model included results from 89,509 control group pupils. The everFSM model 
contained results from 26,799 control group pupils. 


*““The p-value results from a single likelihood ratio test (LRT) across all trial arms. The hypothesis for 
this test was that there is no difference in mean score, adjusted for baseline, between each arm and 


control. 
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Introduction 


Intervention 


The ‘Literacy Octopus’ Dissemination Trial was commissioned by the Education Endowment 
Foundation (EEF) and funded by the EEF, the Department for Education (DfE), and the London Schools 
Excellence Fund. It involved a range of communication approaches to disseminate research evidence 
and evidence-based materials to schools. The passive methods mirrored the passive approaches used 
by providers in the accompanying Evidence-based Literacy Support—the ‘Literacy Octopus’ Trial (Lord 
et al, 2017). In total, the trial consisted of five arms (four intervention arms and one control arm). The 
four providers were each given a randomly-allocated list of 2,500 unique primary schools and 
headteacher contact details so that they could send their materials to these schools by post or email. A 
total of 10,000 schools were sent materials in 2015/2016. There was a single control arm of 2,500 
schools that were not sent any materials. The trial set out to measure the effect on pupil attainment. As 
discussed in the section on background evidence, previous research has suggested that ‘push’ 
dissemination approaches may only have small effects on pupil attainment, but because these methods 
are low-cost they are worth testing at scale. For two of the providers (Arms 2 and 4), the communication 
approach mirrored that used in their passive arm of the Evidence-based Literacy Support — the ‘Literacy 
Octopus’ trial , while Arms 1 and 3 differed slightly (see below). Further details of each approach are 
set out below. 


Trial Arm 1 


The Centre for Evaluation and Monitoring (CEM) at Durham University sent each of their schools the 
evidence-based booklet, /mproving Reading: A Guide for Teachers. In the accompanying trial, schools 
also received materials including a ‘Top Tips’ card (to act as a cue to the booklet), a staffroom poster 
(to act as a cue to the evidenced-based materials and increase awareness of the project throughout 
the school), and monthly classroom activity display posters. These additional materials were not sent 
to schools as part of this Dissemination Trial. 


Trial Arm 2 


The Institute for Effective Education at the University of York sent each of their schools regular print 
issues of the IEE magazine, Better Evidence-based Education (which brings together evidence on 
particular topics and included some literacy specific volumes for schools taking part in the Literacy 
Octopus), its fortnightly email, ‘Best Evidence in Brief’, which summarises new research findings and 
contains links to relevant articles, and access to Evidence 4 Impact, a searchable database of evidence- 
based education intervention programmes (note, neither ‘Best Evidence in Brief’ nor Evidence 4 Impact 
were specifically adapted for the Literacy Octopus). Schools received these materials from February 
2015 to July 2016. 


Trial Arm 3 


ResearchEd and NatCen worked together to provide schools with a one-day conference, using the 
ResearchEd approach, on research evidence for Key Stage 2 (KS2) literacy. The conference involved 
presentations from a number of high profile academics and practitioners. ResearchEd and NatCen sent 
the headteacher in each of their dissemination trial schools an invitation for them and their staff to view 
the website of archived webinar and live materials relating to ResearchEd Literacy Octopus conference 
activity. Note that attendance at the ResearchEd conference was not part of this trial (unlike the 
equivalent passive arm in the accompanying trial where schools were invited to attend a conference— 
see Lord et al., 2017). 


Trial Arm 4 


Teaching How2s runs a subscription website designed for school and further education college CPD. 
Schools and colleges that subscribe to the website gain access to visual guides for particular teaching 
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and CPD techniques. The focus is on supporting teachers to implement evidence-based techniques 
through step-by-step guides; the theory is available separately. The dissemination trial arm was 
managed by the Campaign for Learning on behalf of Train Visual (the organisation that owns Teaching 
How2s). The existing Teaching How2s were updated to include content applicable to primary schools 
and literacy. Schools were given a free subscription to the Teaching How2s website from February 
2015 to July 2016 and were able to access all content, including content developed for the Literacy 
Octopus. 


The control group (Arm 5) 


The control group (Arm 5) of 2,500 randomly-allocated primary schools was not sent any materials by 
these providers during January 2015 to July 2016, but their pupils’ attainment results were included in 
the analysis. 


Background evidence 


Over the past two decades there has been a growing recognition of the value of research in informing 
understanding about what works in ensuring the best educational outcomes for young people. The 
government is keen to develop a more evidence-informed teaching profession and set out a number of 
plans to support this in the White Paper, ‘Educational Excellence Everywhere’ (2016). The current policy 
agenda for an evidence-informed teaching profession has ramifications not only for the teaching 
profession itself, but importantly for producers of research evidence, for example academic and other 
educational research institutes. Indeed, a range of challenges have been noted in relation to research- 
use in education practice including the generalizability of research findings to other contexts and the 
often inaccessible nature of findings to a non-academic audience (for example, CUREE, 2011; 
Goldacre, 2013; Becheikh ef a/., 2009; Durbin and Nelson, 2014; and See et al., 2016). Research 
literature posits that effective knowledge mobilisation requires some translated or interactive support, 
for example guidance on how to apply the evidence in practice (for example, Becheikh et a/., 2009; 
Durbin and Nelson, 2014; Nelson and O’Beirne, 2014; and Sharples, 2013). 


This study explored the nature of large-scale dissemination strategies through relatively inexpensive 
approaches to disseminating research evidence to teachers and schools. It was a large-scale 
effectiveness trial of communication strategies. In order to address some of the challenges highlighted 
above, all of the passive communication methods were selected for trial as they had some element of 
applicability beyond pure dissemination. For example, IEE materials include recommendations for 
practice, How2s include interactive visual guides and encourage teacher peer-to-peer learning, the 
CEM booklet includes top tips for action, and the ResearchEd conference webinar archive included 
video clips from school-presenters that had applied evidence in practice. A study by Tymms and Merrell 
(2006) indicated that combining a widespread ‘push’ dissemination approach with translation of 
research evidence for practical application could be effective: distributing a practical guidance booklet 
to schools on evidence-based strategies to address inattentive, hyperactive, and impulsive pupil 
behaviour led to positive impacts on pupil behaviour and teacher morale. That said, the approaches 
trialled here were all part of the passive dissemination end of the spectrum of other approaches also 
tested out in the sister Evidence-based Literacy Support—the ‘Literacy Octopus’ Trial. Appendix D sets 
out the overall Theory of Change for passive communication, highlighting this inexpensive approach. 


Evaluation objectives 


The Education Endowment Foundation (EEF) commissioned a number of providers to use a range of 
different methods of communicating research findings and evidence to teachers and schools. The EEF 
wishes to determine the impact that these different approaches have on pupils’ literacy. 
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Primary research question 


The primary research question was: What are the effects of different ways of communicating research 
evidence and findings to teachers and schools—and different ways of engaging them with research— 
on pupil attainment for the 2015/2016 Year 6 cohort? 


Secondary research question 


The secondary research question was identical, but focused on the 2016/2017 cohort. 


NFER and the providers attempted to answer this question by running, in parallel, two multi-armed 
RCTs, one which assessed the impact of passive means of communication (this trial) and a second 
which assessed the impact of active and passive means of communication (Evidence-based Literacy 
Support—the ‘Literacy Octopus’ Trial, Lord et al. 2017). 


To determine the impact that different approaches to communicating research evidence have on 
attainment, we measured the development of pupil literacy using KS1 attainment as a baseline and 
KS2 literacy attainment as the outcome measure. 


There were no secondary outcomes specified for this trial. 
Ethical review 


This study obtained approval from NFER’s Code of Practice Group on 3 July 2014. This study planned 
to use de-identified pupil administrative data, that is, use anonymous Pupil Matching Reference (aPMR) 
numbers from the Department for Education (DfE) National Pupil Database (NPD) in relation to KS1 
and KS2 attainment data. There were no special ethical concerns regarding this. 


Schools in the dissemination trial did not need to provide consent to be sent materials. The project team 
and Code of Practice Group gave specific consideration to the nature of the intervention and real-world 
trial—that is, sending materials to schools and then independently analysing NPD data from those 
schools. However, it was agreed with the EEF and providers that providers would insert a footnote into 
their e-materials or websites so that schools could see that the materials they received had been funded 
as part of a trial. 


Project team 


The principal investigator was Dr Ben Styles, Head of NFER’s Education Trials Unit. The day-to-day 
trial manager was Pippa Lord, Senior Research Manager in NFER’s Centre for Evaluation and 
Consultancy. They were supported by NFER statistician Adam Rabiasz and, earlier in the study, by Dr 
Anneka Dawson (who was working at NFER during the school sampling stage). 


The communication strategies were delivered by four organisations or partnerships referred to as 
‘Literacy Octopus providers’. These were the Centre for Evaluation and Monitoring (CEM) at Durham 
University, the Institute for Effective Education (IEE) at The University of York, NatCen working with 
ResearchEd, and Campaign for Learning on behalf of Train Visual (the organisation that owns Teaching 
Howes). 


NFER was responsible for sampling schools, randomising schools, pupil-data analysis, and reporting. 
Literacy Octopus providers were responsible for communicating their passive research materials to 
schools in their arm. NFER put in place a data sharing agreement with each provider to cover how the 
relevant allocated list of schools would be shared (per arm). 


The project was supported and guided by EEF staff Eleanor Stringer, Professor Jonathan Sharples, 
and, from 2016 onwards, Dr Anneka Dawson. 
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Trial registration 


The Literacy Octopus Trial is registered at http://www.isrctn.com/ISRCTN845081 18. This registration 
relates to both the dissemination and evidence-based literacy support trials — Known in the protocol, 
statistical analysis plans and analysis syntax as the Passive and Active Trials respectively. 
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Methods 


Trial design 


The Literacy Octopus Dissemination Trial was a large-scale school randomised effectiveness trial. 
There was no recruitment phase as schools in the dissemination trial were not aware that they were 
part of the dissemination trial in advance of receiving the materials. Materials and emails were simply 
sent by providers, en masse, to schools. Schools were randomised to one of five groups—four 
intervention groups where providers used ‘passive’ means of communicating with schools, or to a 
‘business as usual’ control group. The trial followed a many-to-one comparison scheme (each group 
mean was compared to the control group mean and only the control group mean). 


Participant selection 


The sampling frame of the dissemination trial was defined to include all state primary schools in England 
that had a Year 6, that had not been recruited to the Evidence-based Literacy Support—the ‘Literacy 
Octopus’ Trial, and which were not already in receipt of research communications from the providers. 
From this population, a sample of 12,500 schools was drawn and split into five equal-sized groups, one 
for each provider and one control group. After randomisation, the providers were given their list of 
schools and proceeded to implement their passive communication methods. 


Outcomes measures 


Primary outcome 

The primary outcome was 2015/2016 Key Stage 2 attainment in English. We used the sum of 
READSCORE (scaled score in reading) and GPSSCORE (scaled score in grammar, punctuation, and 
spelling). Writing was not included as this was teacher assessed and is therefore vulnerable to bias. 
Secondary outcome 

The secondary outcome was the same as the above but for the 2016/2017 Year 6 cohort. Results from 
this analysis will be reported in an addendum report in 2018. 


Sample size 


Power calculations performed in the protocol indicated that with four passive arms and a control group, 
each containing 2,500 schools, the estimated minimum detectable effect size (MDES) would be 0.024. 
Sample size calculations were based on KS2 outcomes with KS1 as a baseline using the following 
assumptions 

e average of 34 pupils per cohort per school; 

e ICC =0.15 (reduced from 0.2 through the use of KS1 as a covariate); 

e correlation between KS1 and KS2 = 0.7; 

e power = 80%; and 

e significance = 5%. 


Minimal and unbiased attrition was anticipated as this trial used administrative NPD data only. 
Randomisation 


Simple randomisation was used to allocate 12,500 schools to one of four passive arms or a control 
group (all groups had 2,500 schools). Randomisation was carried out by an NFER statistician. Once 
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randomisation was completed, providers received their list of schools and implemented their passive 
communications strategies with them. The randomisation syntax is included in Appendix C. 


Analysis 
The statistical analysis plan (SAP) for the dissemination trial is available online: 


https://educationendowmentfoundation.org.uk/public/files/Projects/Evaluation_Protocols/SAP_Literacy 
_Octopus_-_Passive.pdf 


Primary intention-to-treat (ITT) analysis 


The primary outcome analysis was ‘intention to treat’. A multilevel model with two levels (school and 
pupil) was used for the analysis to account for the cluster randomisation. The first step was to determine 
if different methods of communicating research had had any significant effect on KS2 attainment. This 
was ascertained by fitting two models to the data: one which had KS2 attainment as the dependent 
variable and KS1 attainment as an independent variable, and a second model identical to the first but 
including four dummy variables that indicated which group an individual was in (the default being 
control). A likelinood ratio test between these models provided a global test for the impact of the passive 
communication strategies used within this trial. 


Analysis continued in terms of presenting effect sizes and confidence intervals for each arm versus 
control, regardless of the outcome of the global test. No post hoc tests were made between the trial’s 
intervention arms as the emphasis of the study was to determine the effectiveness of each individual 
passive approach to the control group. 


When calculating confidence intervals, the two-sided form of Dunnett’s Test (1955) was used to control 
the family-wise error rate. This test is specifically designed to handle the scenario of multiple 
comparisons being made to a common control group and is an exact test (that is, its family-wise error 
rate is exactly equal to alpha) and can be used in balanced and unbalanced designs. 


All multilevel analyses were carried out using the R package nlme. Dunnett’s Test was used to control 
the family-wise error rate using the R package multcomp. 


Imbalance at baseline 


As the primary outcome was available from administrative NPD data and there was no opportunity for 
schools to withdraw it was anticipated that the level of missing data would not exceed 5% at either the 
school or pupil level. However, since pupil-level missing data was at 6%, baseline imbalance was 
explored for ‘percentage everFSM’ and mean KS1 attainment (see ‘Missing data’). 


Missing data 


As this analysis used administrative NPD data, it was anticipated that the number of pupils missing 
would be very small and that any such cases could be excluded from the analysis without risk of bias. 
It was anticipated that the level of missing would not exceed 5% at either the school or pupil level so no 
missing data analysis was planned. 


In the event, however, 103 randomised schools could not be located on administrative NPD data due 
to school closure, merger, or organisational changes such as academisation.! Missing schools were 
spread across the arms. Since these represented just under 1% of all 12,500 schools in the sample, no 
analysis of schools missingness was required. 


1 Both the NPD team and NFER statisticians addressed the issue of URN changes at academisation through the 
use Of link files supplied by DfE. The missing schools are as a result of very recent academisation that ‘fall through 
the gap’ between NPD records and Edubase records. 
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Pupil-level missing data, however—through individuals not having a full complement of test scores at 
baseline and follow-up—accounted for 6% of the total: 28,604 pupils of the 466,799 pupils matched via 
Unique Pupil Numbers (UPNs) to the NPD. We therefore explored the means of the analysed groups 
using school-level ANOVAs on ‘percentage everFSM’ and KS1 school mean scores. We do not believe 
data was missing for any reason that might indicate bias as Literacy Octopus dissemination would not 
interfere with a school’s ability to administer KS2 tests in the usual way. 


On-treatment analysis 


No on-treatment analysis was specified or required for this trial. The only monitoring of delivery was a 
statement from each provider that they had sent their materials as planned. 


Secondary outcome analysis 


Follow-up outcomes using Key Stage 2 English attainment data for the 2016/2017 cohort will be 
analysed and reported in an addendum report in 2018. The same analysis as that outlined in the primary 
intention-to-treat analysis for the 2016 cohort will be used. 


Subgroup analyses 


Sub-group analysis on the primary outcome was carried out on the FSM group only as per the protocol 
and the most recent EEF analysis guidelines—that is, whether a pupil has ever received free school 
meals (‘everFSM’).? This was done using a model identical to the primary outcome model but including 
everFSM and everFSM‘intervention as covariates too. Analysis proceeded as per the original primary 
outcome modelling, that is, the first model was identical to the primary outcome model but with everFSM 
as a covariate. The second model contained a further eight covariates: four dummies for randomised 
group and four interaction terms. A likelihood ratio test was performed between the two models. 


Asecond model was also run including only everFSM children, as per EEF analysis guidelines. Analysis 
conformed to the original primary outcome model. 


Effect size calculation 


All effect sizes were calculated using total variance from a multilevel model, without covariates, as the 
denominator—equivalent to Hedges’ g. Confidence intervals for each effect size were derived from 
Dunnett’s Test (see above) to take into account multiple comparisons. Dunnett’s Test applied to the 
model coefficients themselves. These were then converted to effect size confidence intervals using the 
same formula as the effect size itself. 


Implementation and process evaluation 


There was no implementation and process evaluation (IPE) specified for this trial—except for providers 
to confirm that they had indeed sent their materials and emails as planned. 


Costs 


There was no cost evaluation specified for this trial. We did not collect any cost data from schools and 
the EEF do not require providers’ costs to be reported. 


? We used the NPD EVERFSM_ALL variable which entails a flag to indicate if a pupil has ever been recorded as 
eligible for free school meals on the census day in any spring census up to the pupil's current year (not including 
nursery). 
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Timeline 
Table 2: Timeline 


Date Activity 


Summer term 2014 Write, agree, and register protocol. 


Devise sample strategy (after Evidence-based literacy support — the 
‘Literacy Octopus’ trial sampling frame). 


Sample schools. 
Randomly allocate schools to one of five groups (Dec 2014). 


Spring term 2015 Providers send intervention materials to schools. 
Autumn term 2016 Write and agree SAP. 


Autumn term 2014 


Intention-to-treat analysis using NPD data (primary outcome 2015/2016). 
Spring term 2017 Sub-group analysis (everFSM). 
Write report. 


Autumn term 2017 Publish main report. 
Further activities to be undertaken (2017-2018) 


Intention-to-treat analysis using NPD data (follow-up 2016/2017). 
Write addendum report. 


Summer term 2018 Publish addendum report . 


NFER timetable for the Literacy Octopus Dissemination Trial. Evaluation activity shown in normal font. Provider/intervention 
activity shown in italics. 


Spring term 2018 
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Impact evaluation 


Participants 


Figure 1 provides details of the participant flow through each stage of the ‘Literacy Octopus’ 
Dissemination Trial. 


As Figure 1 illustrates, 12,500 schools were randomised for the this trial of which 103 could not be 
matched from our randomised list to the NPD dataset provided by the DfE NPD team. Reasons were 
entirely down to changes in school organisation such as closures, mergers, or academisation. There 
were no biased reasons. This represented just under 1% of the randomised schools and hence no 
further analysis was required to address this. 


In terms of pupil attainment results used in the trial, we were unable to match KS1 and KS2 cases for 
28,604 of the 466,799 pupil results in the trial—6% of cases. ANOVAs on baseline means for KS1 
pupils and everFSM pupils, reported under Table 4, suggested that this level of missingness was 
unbiased. 


Table 3 provides details of minimum detectable effect sizes at different stages in the trial. 


Figure 1: Participant flow diagram—‘Literacy Octopus’ Dissemination Trial 


Randomised 
(school n = 12,500; pupil n@ = 466,779) 


Arm 1 Arm 2 Arm 3 Arm 4 Arm 5 
CEM IEE ResearchEd How2s Control 
(school n = (school n = (school n = (school n = (school n = 
2,500; pupil n? 2,500; pupil n 2,500; pupil n = 2,500; pupil n 2,500; pupil n 
= 93,864) = 92,465) = 93,218) = 95,378) 


analysed analysed analysed analysed analysed 
n=19 n=21 n=26 n=18 n=19 


Arm 1 Arm 2 Arm 3 Arm 4 Arm 5 
CEM IEE ResearchEd How2s Control 
(school n = (school n = (school n = (school n = (school n = 
2,481; pupil n 2,479; pupil n 2,474; pupil n 2,482; pupil n 2,481; pupil n 
= 88,088) = 86,742) = 86,155) = 87,701) = 89,509) 


4 Determined from the match to NPD, using data from the 12,397 schools that could be matched to NPD. 
> Could not be matched back to NPD for reasons such as school closure or merger: 103 of the 12,500 
randomised schools could not be matched to NPD. 
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Table 3: Minimum detectable effect size (MDES) at different stages 


n schools (n = 
(F-Ye ae lai) 


n pupils (n = 
each arm) 


Correlation 
between 
pre-test & 
post-test 


=} [oted diate 
stratification 
Or pair 
matching 


Power Alpha 


12,500 (2,500) 
Protocol 0.70 0.15 n/a 80% 0.05 0.024 
437,500 (85,000) 
12,500 (2,500) 
Randomisation 466,779 0.70 0.15 n/a 80% 0.05 0.024 
(minimum 91,854) 
12,397 (minimum 
Analysis (i.e. 2,474) 
available pre- 0.68 0.14 n/a 80% 0.05 0.024 
and post-test) 438,195 
(minimum 86, 155) 


Source: NFER Literacy Octopus Dissemination Trial, 2016 
Pupil characteristics 


Randomised schools that could not be matched to the NPD for reasons such as closures and mergers 
accounted for less than 1% of the original 12,500. This was a very small fraction of a large number of 
schools and occurred for reasons very unlikely to be due to bias. We have therefore omitted school- 
level factors from the baseline comparison table. Pupil-level missing data, however, through individuals 
not having a full complement of test scores at baseline and follow-up, accounted for 6%. We have 
therefore included two comparisons of school means on analysed groups (percentage everFSM and 
KS1) in Table 4. To interpret this table it is helpful to know that the overall standard deviation of the 
percentage everFSM and KS1 school mean measures were 20.8 and 2.55, respectively. 


Table 4: Baseline comparison 


CEM: advice 
booklet only 
(Arm 1) 


Research Ed: 
invitation to webinar 
materials only (Arm 3) 


Teaching 
How2s: login 
only (Arm 4) 


IEE: evidence 
material only 
(Arm 2) 


Pupil-level 


NZ-Varle)(:} lefey alice) | 


(analysed 
at school- n n n n n 
EAZ=1)) (missing) mean (missing) mean (missing) mean (missing) mean _ (missing) | mean 
Percentage 2,481 2,479 2,474 2,482 2,481 
everFSM (19) 29.1 (21) 29.2 (26) 28.2 (18) 29.3 (19) 29.6 
a 2,481 2,479 2,474 2,482 2,481 
pee (19) 31.12 (21) 31.14 (26) 31.18 (18) 31.10 (19) 31.14 


Source: NFER Literacy Octopus Dissemination Trial, 2016 

An ANOVA on ‘percentage everFSM’ (df = 4, F = 1.495, p = 0.201) suggested that the differences we 
see were due to chance. The ANOVA on KS1 school means (df = 4, F = 0.335, p = 0.854) also 
suggested that any differences were due to chance. In terms of a pre-test effect size on analysed 
groups, the largest was -0.017 between Teaching How2s and control (a small difference when 
interpreting effect sizes). 
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Outcomes and analysis 


Primary outcome and sub-group analysis 


The main result was from a likelihood ratio test between a multi-level model containing just baseline 
and one containing baseline plus all the intervention groups as covariates. This yielded a likelihood ratio 
of 3.45 (p = 0.48) meaning that we do not reject the null hypothesis and there is no evidence of any 
effect of the various interventions. In order to present effect sizes, confidence intervals were calculated 
using Dunnett’s Test to correct for multiple inference: the results are presented in Table 5a and Figures 
2a and 2b; these show results for the primary outcome (KS2 literacy attainment for the 2015/2016 
cohort) and for the everFSM subgroup. 


Table 5a: Primary analysis—effect size versus control (Dunnett’s multi-level confidence 
intervals); numbers of schools (n) and numbers of pupils (N) 


Research Ed: 
CEM: advice IEE: evidence invitation to Teaching How2s: 
booklet only material only webinar only login only Lee} nl ice) 


(Arm 1) (Arm 2) (Arm 3) (Arm 4) (Arm 5) 


nN ES nN ES nN ES nN ES nN 
Outcome (missing) | (95% Cl) (missing) (95% Cl) (missing) | (95%Cl) (missing) (95% Cl) (missing) 


n 2,481 n 2,479 n 2,474 n 2,482 n 2,481 
KS2 (19) 0.01 (21) 0.01 (26) 0.01 (18) 0.01 (19) 
English 88,088 (-0.01, | N86,742 (0.01, | N86,155 | (0.01, | N87,701 | (0.01, | N89,509 
(5,776) 0.03) (5,723) 0.03) (5,699) 0.03) (5,517) 0.03) (5,869) 
Ks2 n 2,346 n 2,335 n 2,343 n 2,354 n 2,361 
English (154) 0.00 (165) -0.01 (157) 0.00 (146) 0.00 (139) 
des N 26,319 (0.03, N26,243 (0.04, N25,059 (09.03, | N26,489 (0.03, | N26,799 
(1,708) 0.03) (1,764) 0.02) (1,664) 0.03) (1,709) 0.03) (1,725) 


Source: NFER Literacy Octopus Dissemination Trial, 2016 
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Figure 2a: Primary analysis—effect size versus control (Dunnett’s multi-level confidence 
intervals) 


Effect size between control group 
and trial arm 


CEM Advice booklet ——-_—___-—______} 
! 
! 
IEE Evidence material (++) 
! 
Research Ed invitation to webinar a 
! 
! 
Teaching How2s login A>) 


->——— st tp 
-0.035 -0.021 -0.007 0.007 0.021 0.035 


Effect size 


Figure 2b: FSM-only analysis—effect size versus control (Dunnett’s multi-level confidence 
intervals) 


Effect size between control group 
and trial arm 


CEM Advice booklet 


IEE Evidence material 


' 
' 
Research Ed invitation to webinar (+) 


! 
! 
' 
' 
Teaching How2s login (——_e—— 


-0.050 -0.025 0.000 0.025 0.050 


Effect size 
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We also explored raw unadjusted means using Key Stage 2 English attainment data for all pupils in 
dissemination trial schools in the NPD 2015/2016 KS2 dataset (that is, those with KS2 scores, but prior 
to the analysed KS1-KS2 match used in the main analysis reported in Table 5a). The results are shown 
in Table 5b enabling the reader to see the raw results prior to complex modelling. Missing data relates 
to the number of pupils missing KS2 data on the NPD list provided by DfE. Note: the participant flow 
diagram provides numbers of pupils in the main primary outcome analysis (that is, those shown in Table 
5a). 


Table 5b: Raw means—unadjusted confidence intervals, numbers of pupils (N) 
Research Ed: 


CEM: advice IEE: evidence invitation to Teaching How2s: 
booklet only material only webinar only login only 
(Arm 1) CNG) Ng) (Arm 4) Control (Arm 5) 
nN ES nN ES nN nN ES nN ES 
(O]Ti CoCo) it{- Mmmm UTES) 1¢ ) mm (Lohr O71) (LAT CSoi Lae) i (cy-o)e(-O8 C2 ) NN CLULCSSJL11¢ ) IN @]U] Cove) ga(oMm (LUL(SJ101¢ ) I (Jor ©2) I (LLL) Le ) Im (Jo br/- @2 
— 91,842 20598 90,395 | 205.98  g9662 | 20619 | 91,216 | 206.11 93,341 208-95 
English (2,022) (205.87, (2,070) _| (205.86, (2092) | (206.07, (2,002) (205.99, (2,037) (200.84, 
206.10) 206.10) 206.30) 206.23) 206.07) 
KS2 
English 27,090 | 200-32 57948 | 200.25 | 25,754 200.55 27,294 | 200-45 57699 | 200.63 
(everFSM = (937) (200.08, ggg) (200.01, (99) (200.31, (go4) (200.22, | gis) (200.40, 
only) 200.55) 200.49) ( 200.79) 200.69) 200.86) 


Source: NFER Literacy Octopus Dissemination Trial, 2016 
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Process evaluation 


All providers confirmed that they sent their materials as planned to the headteacher contacts at each of 
their 2,500 allocated schools. The ‘Literacy Octopus’ Dissemination Trial did not require any monitoring 
of sent or received items. 
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Conclusion 


Key conclusions 


1. The project found no evidence that Literacy Octopus passive dissemination interventions 
improved pupils’ attainment in English Key Stage 2 scores compared with the control group. 
The five padlock security rating means we have very high confidence in this result. 


2. These findings suggest that simply disseminating research summaries and evidence-based 
resources to schools is not an effective way for research organisations to support schools to 
improve pupil outcomes. 


3. Itis likely that these materials formed a small part of the information received by schools during 
this time. It is possible that if schools had support to navigate and prioritise that information, 
greater impact could be achieved. Alternatively, schools may need more support in transforming 
such materials into actual change in the classroom. The accompanying Evidence-based 
Literacy Support—the ‘Literacy Octopus’ Trial helps us to assess this. 


4. The evaluator will analyse the following year’s Key Stage 2 results from these schools to explore 
if there is a longer-term effect of the interventions. 


Interpretation 


There was no evidence that any of the Literacy Octopus passive communication approaches impacted 
on pupil attainment in literacy. This was the case for all pupils, and for the everFSM subgroup of pupils. 


The extent to which schools engaged with or used the materials in this dissemination trial is not known 
as it was not monitored. However, we can say from this trial that attempting to reach a large number of 
schools and teachers through fairly light-touch and inexpensive approaches does not impact on 
teaching and learning in a measurable way according to pupil attainment outcomes. This is not to say 
that schools did not use or engage with the materials in any way: we know from the Evidence-based 
Literacy Support — the ‘Literacy Octopus’ trial that just under three-fifths of schools did engage, at least 
to some extent, in passive arm activity. However, any engagement, or otherwise, did not result in 
impacts on pupil attainment. 


The findings answer the main question in the Theory of Change which asked if it might be worth sending 
inexpensive materials to a large number of participants as a way of initiating practice-change and 
improving learning. For researchers, and indeed schools, the findings suggest that this is not a 
worthwhile effort, at least, not on its own. Furthermore, even where materials include some elements 
of applicability that are known to be effective (Durbin and Nelson, 2014; Sharples, 2013), as was the 
case in all of the Literacy Octopus providers’ materials (for example, they were well structured and 
included examples for practice), simple passive dissemination is insufficient to make a difference to 
teaching and learning. 


It is also worth noting that all of the communication in this trial was direct to the headteacher of each 
school. Research literature indicates that senior leader buy-in is key to school-wide knowledge 
mobilisation (Nelson and O’Beirne, 2014; Speight ef a/, 2016). But again, despite targeting 
communication at the most relevant level, simple ‘push’ dissemination approaches as trialled in this 
study seem insufficient to affect change (at least, in terms of pupil attainment outcomes as measured 
on this trial). Furthermore, headteachers are likely to be targeted by other information too, and may 
need support to navigate this. 


Control school ‘business as usual’ was not monitored on this dissemination trial. However, it is clear 
from policy and practice changes during the trial period (2014-2016) that schools are increasingly being 
encouraged to make evidence-informed decisions (DfE, 2016) and it is highly likely that all schools in 
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this trial were receiving research and commercial product marketing information as a matter of course. 
The Literacy Octopus materials were probably light touch additions to a schools’ usual information 
receipts. The question remains—What level of supported research engagement activity would help 
schools to use and engage with research, to improve pupil outcomes? 


Limitations 


The primary outcome analysis for this study had no limitations in terms of bias and precision as it used 
administrative NPD data and there were only minor incidences of missing data: just under 1% of schools 
(103) could not be found on the NPD-supplied data—purely for reasons such as school academisation, 
amalgamation, or school name changes. Pupil-level missing data, however, through individuals not 
having a full complement of test scores at baseline and follow-up, accounted for 6%, but this was not 
found to affect balance at baseline. 


Because of the scale of the dissemination trial, involving 12,500 schools—over 80% of all primary 
schools in England—the results can be generalized to the whole school population in England. 
However, study limitations include the fact that actual usage and engagement with the materials was 
not monitored, neither was control group ‘business as usual’. That said, we know from the existing 
school market landscape that all schools receive posted and emailed materials as a matter of course 
and hence it seems entirely reasonable not to require monitoring of this. 


Future research and publications 


The impact on pupils’ literacy attainment in Key Stage 2 for the successive cohort (2016/2017) will be 
analysed and reported in an addendum report in 2018. This reflects the Theory of Change—that 
changes in pupil attainment may take time to surface making it pertinent to explore two cohorts’ worth 
of attainment data. 


The ‘Literacy Octopus’ Dissemination Trial had no impact on attainment. The materials were light touch 
and used passive dissemination methods. The findings raise two key questions: first, how to better 
attract schools’ attention to the value of research evidence among the surfeit of disseminated 
information they receive, and second—in combination with findings from the Evidence-based Literacy 
Support — the ‘Literacy Octopus’ trial —to consider what level of additional supported research 
engagement activity would help schools to use and engage with research in order to improve pupil 
outcomes. 


Education Endowment Foundation 21 


Literacy Octopus Dissemination Trial 


References 


Becheikh, N., Ziam, S., Idrissi, O., Castonguay, Y. and Landry, R. (2009) ‘How to improve knowledge 
transfer strategies and practices in education? Answers from a systematic literature review’, Research 
in Higher Education Journal, 7, pp. 1-21. 


Centre for the Use of Research and Evidence in Education (2011) Report of Professional Practitioner 
Use of Research Review: Practitioner Engagement in and/or with Research, Coventry: CUREE. 
Available: http://www.curee.co.uk/files/publication/%5bsite- 
timestamp%5d/Practitioner%20Use%200f%20 Research%20Review%20- 
%20FINAL%2011_02_11.pdf [accessed 29 March 2017]. 


Department for Education (DfE) (2016) Educational Excellence Everywhere. Available: 
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/508447/Educational_Ex 
cellence_Everywhere.pdf [accessed 29 March 2017]. 


Durbin, B. and Nelson, J. (2014) Why Effective use of Evidence in the Classroom Needs System-wide 
Change (NFER Thinks: What the Evidence Tells Us), Slough: NFER. Available: 
https://www.nfer.ac.uk/publications/99942/ [29 March 2017]. 


Goldacre, B. (2013) Building Evidence into Education. Available: 
http://media.education.gov.uk/assets/files/pdf/b/ben%20goldacre%20paper.pdf [accessed 29 March 
2017]. 


Lord, P., Rabiasz, A., Roy, P., Harland, J., Styles, B., and Fowler, K. (2017) Evaluation of evidence- 
based literacy support — the ‘Literacy Octopus’ trial. London: Education Endowment Foundation. 
Available: https://educationendowmentfoundation.org.uk/projects-and-evaluation/projects/the-literacy- 
octopus-communicating-and-engaging-with-research/ 


Nelson, J. and O’Beirne, C. (2014) Using Evidence in the Classroom: What Works and Why? Slough: 
NFER. Available: https:/Awww.nfer.ac.uk/publications/IMPA01/IMPA01.pdf [accessed 29 March 2017]. 


Poet, H., Mehta, P. and Nelson, J. (2015) Research Use in Schools - Survey, Analysis and Guidance 
for Evaluators, Slough: NFER (unpublished report). 


See, B. H., Gorard, S. and Siddiqui, N. (2016) ‘Teachers’ use of research evidence in practice: a pilot 
study of feedback to enhance learning’, Educational Research, 58 (1), pp. 56—72. 


Sharples, J. (2013) Evidence for the Frontline, London: Alliance for Useful Evidence. Available: 
http://www.alliance4usefulevidence.org/assets/EVIDENCE-FOR-THE-FRONTLINE-FINAL-5-June- 
2013.pdf [accessed 29 March 2017]. 


Speight, S., Callanan, M., Griggs, J. and Caragena-Farias, J. (2016) Rochdale Research into 
Practice: Evaluation report and executive summary. Available: 
https://education.gov.scot/improvement/Documents/sac46-eef-research-into-practice.pdf [accessed 
12 April 2017]. 


Education Endowment Foundation 22 


Literacy Octopus Dissemination Trial 


Appendix A: EEF cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention over 
three years. More information about the EEF’s approach to cost evaluation can be found here. Cost 
ratings are awarded as follows: 


Cost rating | Description | 


EELEEL Very low: less than £80 per pupil per year. 
£ELEL Low: up to about £200 per pupil per year. 


£EEEL Moderate: up to about £700 per pupil per year 
LELEE High: up to £1,200 per pupil per year. 
oe | Very high: over £1,200 per pupil per year. 
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Appendix B: Security classification of trial findings 


atcLalare| Criteria for rating mutta Adjust 


4a Fair and clear quasi- 
experimental design for 
comparison (e.g. RDD) with | jwpES~< : Adjustment 
appropriate analysis, or 03 11-20% for Balance 
experimental design with [0] 
minor concerns about validity 
3@ |Well-matched comparisoni | £|  &|| fo [fo 
(using propensity score 
matching, or similar) or] wpES~< 
experimental design with 0.4 21-30% 
moderate concerns about 
i ee re eee Lae ee Adjustment |__| 
- Weakly matched comparison fed 
r experimental design with | MDES < 6 ao 
oe 05 31-40% validity 
major flaws : 
_ olf 
18 ; 
Comparison group with poor 
or no matching (E.g. volunteer | MDES < 41-50% 
versus others) 0.6 
Os 
No comparator MDES> | over 50% 
0.6 


e Initial padlock score: lowest of the three ratings for design, power and attrition = 5 padlocks 
e Reason for adjustment for balance (if made): N/A 

e Reason for adjustment for threats to validity (if made): N/A 

e Final padlock score: initial score adjusted for balance and internal validity = 5 padlocks 
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Appendix C: Randomisation Syntax 


title 'EEFA: Passive Trial’. 
set printback=on. 


GET DATA /TYPE=XLSX 
/FILE='K:\EEFA\FINAL list for passive trial.xlsx' 
/SHEET=name 'FINAL passive list 221214' 
/CELLRANGE=tfull 
/READNAMES=on 


/ASSUMEDSTRWIDTH=32767. 


* Check for duplicate nfer_no. 
sort cases by nfer_no. 
match files file=*/first=f1/last=I1/by nfer_no. 


cross f1 by I1. 


* Check for duplicate contact_id. 
sort cases by contact_id. 
match files file=*/first=f2/last=I2/oy contact_id. 


cross f2 by 12. 


set rng=mt, mtindex=12. 

compute randome=rv.uniform(0,1). 
print formats random (F15.13). 

exe. 

sort cases by random. 

list vars random/cases=from 1 to 20. 
“Freq random. 


compute lineno=$casenum. 
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exe. 


numeric group (F2.0). 

if $¢asenum le 2500 group=1. 

if $easenum ge 2501 and $casenum le 5000 group=2. 
if $¢asenum ge 5001 and $casenum le 7500 group=3. 
if $¢asenum ge 7501 and $casenum le 10000 group=4. 


if $¢asenum ge 10001 group=5. 


freq group. 


sort cases by group. 


add value labels group 1 '1: CEM - Advice booklet only' 2 '2: IEE - Evidence material only’ 3 '3: 
Research Ed - Invitation to webinar only' 


4'4: Teaching Howes - Login only’ 5 '5: Control group". 


SAVE TRANSLATE OUTFILE='K:\EEFA\Passive trial randomisation.xlsx' 
/TYPE=XLSX 
/VERSION=12 
/MAP 
/REPLACE 
/FIELDNAMES 
/CELLS=VALUES 


/KEEP=Contact_ID NFER_No group Region Northwithformula Southwithformula. 
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Appendix D: A Theory of Change 


The Theory of Change for the Literacy Octopus Dissemination Trial posits the question whether 
inexpensive ‘push’ dissemination approaches will have an effect. Inexpensive materials can be sent to 
a large number of schools/teachers to test out this question. 
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